Papers with NLP model
The economic trade-offs of large language models: A case study (2023.acl-industry)
Copied to clipboard
Kristen Howell, Gwen Christian, Pavel Fomitchov, Gitit Kehat, Julianne Marzulla, Leanne Rolston, Jadin Tredup, Ilana Zimmerman, Ethan Selfridge, Joseph Bradley
| Challenge: | Large Language Models (LLMs) are a natural fit for contact-based customer service, but their efficacy must be balanced with the cost of training and serving them. |
| Approach: | They propose a cost framework for evaluating an NLP model’s utility for the enterprise as a function of the usefulness of the responses that they generate. |
| Outcome: | The proposed model can be used to help human agents handle complex customer service calls and can be modified to improve their performance. |
Empirical Studies of Institutional Federated Learning For Natural Language Processing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | federated learning is a promising ideology to unite isolated datasets for machine learning problems. |
| Approach: | They propose to use federated natural language processing networks to train a popular NLP model with applications in sentence intent classification. |
| Outcome: | The proposed model is sensitive to imbalanced data load and tested against a federated model under imbalanced datasets. |
Two-Step Classification using Recasted Data for Low Resource Settings (2020.aacl-main)
Copied to clipboard
Shagun Uppal, Vivek Gupta, Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, Amanda Stent
| Challenge: | Existing studies on NLP models focus on high resource languages like English, but there are only two datasets for Hindi. |
| Approach: | They propose a novel two-step classification method which uses textual-entailment predictions for classification task. |
| Outcome: | The proposed method improves classification performance by using a joint-objective for classification and textual entailment. |
Towards Improving Adversarial Training of NLP Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for generating NLP adversarial examples involve combinatorial search and expensive sentence encoders for constraining the generated instances. |
| Approach: | They propose to use vanilla adversarial training to train NLP models using a word substitution attack optimized for vanilla adversary training. |
| Outcome: | The proposed approach improves model performance and standard accuracy and can defend against other types of word substitution attacks. |
Self-Adapted Utterance Selection for Suicidal Ideation Detection in Lifeline Conversations (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for identifying suicidal ideation in phone conversations are difficult to use because of their long duration and noisy nature. |
| Approach: | They propose a self-adaptive approach that identifies the most critical utterances that the NLP model can more easily distinguish. |
| Outcome: | The proposed approach outperforms the baseline models in overall performance with an F score of 66.01% and significantly higher F-score in detecting the most dangerous cases. |
Pre-train or Annotate? Domain Adaptation with a Constrained Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that pre-training in-domain language models can boost performance when adapting to a new domain. |
| Approach: | They propose to combine annotation and pre-training to maximize performance under budget constraints. |
| Outcome: | The proposed approach is based on the annotation cost of three procedural text datasets and pre-training cost of 3 in-domain language models. |
Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. |
| Approach: | They propose a method to quantify the severity of shortcut reasoning by leveraging out-of-distribution data. |
| Outcome: | The proposed method quantifies the severity of the discovered shortcut reasoning using out-of-distribution data. |
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2022.naacl-main)
Copied to clipboard
| Challenge: | a pretrained model is optionally adapted through domain-specific pretraining, followed by task-specific finetuning. |
| Approach: | They establish a suite of eight tasks across different domains to quantify the effects of temporal misalignment in modern NLP systems. |
| Outcome: | The proposed tasks are based on eight domains and periods of time spanning five years or more and show that they have stronger effects than previous studies. |
MBTI Personality Prediction for Fictional Characters Using Movie Scripts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing NLP models cannot predict character's personality types based on text classifications . character comprehension is the cornerstone of understanding stories in psychology and education. |
| Approach: | They propose a benchmark to predict movie character's MBTI or Big 5 personality types based on the narratives of the character. |
| Outcome: | The proposed model outperforms existing models in the task and is more accurate than random guesses. |
Distilling Structured Knowledge for Text-Based Relational Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-based relational reasoning models lack a symbolic representation of text . performance gap between NLP models and structured models remains . |
| Approach: | They first pre-train a GNN on a reasoning task using structured inputs and then incorporate its knowledge into an NLP model. |
| Outcome: | The proposed model improves on two state-of-the-art NLP models on 13 different inductive reasoning datasets from the CLUTRR benchmark. |
We Need to Talk About Reproducibility in NLP Model Comparison (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that standard splits produce low reproducible and unreliable conclusions . reproducibility of empirical experimental conclusions is a problem in NLP domain . |
| Approach: | They propose to transform the reproducibility of a model comparison into a probabilistic function . they propose to use a regularized corpus splitting strategy to estimate the model's performance . |
| Outcome: | The proposed estimator achieves a high SNR and significantly increases reproducibility. |
Predicting generalization performance with correctness discriminators (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models estimate accuracy of models on unlabeled test data, but they hide their own uncertainty. |
| Approach: | They propose a model that establishes upper and lower bounds on the accuracy without requiring gold labels for the unseen data. |
| Outcome: | The proposed model establishes upper and lower bounds on accuracy without requiring gold labels for the unseen data. |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |